Asynchronous optimization for sequence training of neural networks

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, by a first sequence-training speech model, a first batch of training frames that represent speech features of first training utterances; obtaining, by the first sequence-training speech model, one or more first neural network parameters; determining, by the first sequence-training speech model, one or more optimized first neural network parameters based on (i) the first batch of training frames and (ii) the one or more first neural network parameters; obtaining, by a second sequence-training speech model, a second batch of training frames that represent speech features of second training utterances; obtaining one or more second neural network parameters; and determining, by the second sequence-training speech model, one or more optimized second neural network parameters based on (i) the second batch of training frames and (ii) the one or more second neural network parameters.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation (and claims the benefit ofpriority under 35 U.S.C. § 120) of U.S. patent application Ser. No.15/910,720, filed Mar. 2, 2018, which claims the benefit of priorityunder 35 U.S.C. § 120) of U.S. patent application Ser. No. 14/258,139,filed Apr. 22, 2014, which claims the benefit of U.S. Provisional PatentApplication No. 61/899,466, filed Nov. 4, 2013. The entire contents ofthe prior applications are incorporated by reference herein.

TECHNICAL FIELD

This specification generally relates to training speech models forspeech recognition.

BACKGROUND

In some instances, it may be useful to sequence-train a speech model torecognize a spoken phrase that includes words uttered in a sequence.Sequence training requires computationally intensive processing comparedto frame-level training. Asynchronous optimization may allow sequencetraining of multiple speech models to update model parameters inparallel and enable scalable sequence training.

SUMMARY

According to one innovative aspect of the subject matter described inthis specification, a speech model based on a neural network may betrained using training utterances of a set of training speakers. Eachtraining utterance may include a sequence of spoken words. Multiplespeech models may be asynchronously trained in parallel, where theasynchrony may allow each speech model to be locally optimized.

In general, one innovative aspect of the subject matter described inthis specification can be embodied in methods that include the actionsof obtaining, by a first sequence-training speech model, a first batchof training frames that represent speech features of first trainingutterances, obtaining, by the first sequence-training speech model, oneor more first neural network parameters, determining, by the firstsequence-training speech model, one or more optimized first neuralnetwork parameters based on (i) the first batch of training frames and(ii) the one or more first neural network parameters, obtaining, by asecond sequence-training speech model, a second batch of training framesthat represent speech features of second training utterances, where theobtaining of the second batch of training frames by the secondsequence-training speech model is independent of the obtaining of thefirst batch of training frames by the first sequence-training speechmodel, obtaining one or more second neural network parameters, where theobtaining of the second neural network parameters by the secondsequence-training speech model is independent of (i) the obtaining ofthe first neural network parameters by the first sequence-trainingspeech model and (ii) the determining of the one or more optimized firstneural network parameters by the first sequence-training speech model,and determining, by the second sequence-training speech model, one ormore optimized second neural network parameters based on (i) the secondbatch of training frames and (ii) the one or more second neural networkparameters, where the determining of the one or more optimized secondneural network parameters by the second sequence-training speech modelis independent of the determining of the one or more optimized firstneural network parameters by the first sequence-training speech model

These and other embodiments may each optionally include one or more ofthe following features. Obtaining a first batch of training frames thatrepresent speech features of first training utterances may includeobtaining, by a first decoder associated with the firstsequence-training speech model, an utterance of the first trainingutterances, determining, by the first decoder, a reference scoreassociated with the utterance, determining, by the first decoder, adecoding score associated with the utterance, and determining, by thefirst decoder, a training frame representing acoustic characteristics ofthe utterance based on the reference score.

Determining a reference score associated with the utterance may includeobtaining, by the first decoder, the one or more first neural networkparameters, obtaining, by the first decoder, one or more referenceparameters representing a true transcription of the utterance, anddetermining, by the first decoder, a reference lattice based on (i) theutterance of the first training utterances, (ii) the one or more firstneural network parameters, and (iii) the one or more referenceparameters representing the true transcription of the utterance.

Obtaining a first batch of training frames that represent speechfeatures of first training utterances may include determining, by thefirst decoder, a decoding score associated with the utterance, and wheredetermining the training frame representing the acoustic characteristicsof the utterance may include determining, by the first decoder, thetraining frame representing the acoustic characteristics of theutterance based on the reference score and the decoding score

Determining a decoding score associated with the utterance may includeobtaining, by the first decoder, the one or more first neural networkparameters, obtaining, by the first decoder, one or more decodingparameters representing a candidate transcription of the utterance, anddetermining, by the first decoder, a decoding lattice based on (i) theutterance of the first training utterances, (ii) the one or more firstneural network parameters, and (iii) the one or more decoding parametersrepresenting the candidate transcription of the utterance. The trainingframe may include an outer gradient representing a difference betweenthe reference score and the decoding score.

The actions may include before obtaining, by the first sequence-trainingspeech model, the first batch of training frames, obtaining an initialbatch of training frames, where each training frame represents asequence of utterances spoken by a training speaker, pseudo-randomlyselecting candidate training frames from the initial batch of trainingframes, and generating the first batch of training frames using thepseudo-randomly selected candidate training frames.

Determining one or more optimized first neural network parameters mayinclude obtaining an auxiliary function representing an approximation ofa training objective function of the first sequence-training speechmodel, and determining the one or more optimized first neural networkparameters using the auxiliary function.

The training objective function may be a maximum likelihood (ML)objective function, a maximum mutual information (MMI) objectivefunction, a minimum phone error (MPE) objective function, or astate-level minimum Bayes risk objective function.

Determining one or more optimized first neural network parameters mayinclude updating a last hidden layer of a neural network of the firstsequence training speech model, using (i) the first batch of trainingframes and (ii) the one or more first neural network parameters todetermine the one or more optimized first neural network parameters.

Determining one or more optimized first neural network parameters mayinclude updating a plurality of hidden layers of a neural network of thefirst sequence training speech model, using (i) the first batch oftraining frames and (ii) the one or more first neural network parametersto determine the one or more optimized first neural network parameters.

Advantageous implementations may include one or more of the followingfeatures. Asynchronous optimization allows multiple speech models to betrained in parallel, which allows the system to scale the sequencetraining using large data sets. Shuffling of training utterance inputallows randomization of training utterances that may provide bettermodel parameters. Model optimization using an auxiliary function allowsa speech model to be optimized locally to increase efficiency.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other potentialfeatures and advantages will become apparent from the description, thedrawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of an example system for asynchronousoptimization of multiple speech models in parallel.

FIG. 1B is a block diagram of an example system for generating a framebatch for training a speech model.

FIG. 1C is a block diagram of an example system for training a neuralnetwork of a speech model.

FIG. 2 is a flow chart illustrating an example process for determining aframe for training a speech model.

FIG. 3 is a flow chart illustrating an example process for determining areference score in a decoder.

FIG. 4 is a flow chart illustrating an example process for determining adecoding score in a decoder.

FIG. 5 is a flow chart illustrating an example process for generatingmodel parameter gradients for a speech model.

FIGS. 6A and 6B are diagrams that illustrate examples of word lattices.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

A speech recognition process is the task of transcribing one or morewords in a phrase spoken by a user. In general, a computer-implementedspeech model may be trained to perform the speech recognition process.In some implementations, a speech model may be implemented using aneural network, where the neural network may be optimized throughsequence training. One goal of sequence training is to minimize worderror, so the speech model may recognize the phrase as a whole topreserve the context of the phrase. This is in contrast with frame-leveltraining, where the objective may be to minimize phonetic error withoutconsidering the context of the phrase.

Neural networks may be established for acoustic modeling in speechrecognition. In some implementations, a framework to incorporate aneural network into a Hidden Markov Models (HMM)-based decoder mayinclude the hybrid approach. In some other implementations, theframework to incorporate a neural network into a HMM-based decoder mayinclude the tandem approach.

In some implementations, stochastic gradient descent (SGD) may be usedas the optimization procedure for frame training of neural networks.However, the formulation of SGD makes the optimization a sequentialprocess that is a challenge to parallelize, and hence creates issues inscaling to large datasets. In general, for stable SGD optimization, theobservations should be randomized (shuffled). This is in contrast withthe computation of the gradient which is utterance-derived and hencesequential in nature and a challenge to shuffle. In general, batchoptimization schemes are a more natural match for this type ofobjective. Asynchronous training of multiple speech models in parallelmay enable easier scalability in the sequence training process usinglarge training datasets,

FIG. 1A is a block diagram of an example system 100 that canasynchronously train multiple speech model replicas in parallel, whereeach speech model is implemented based on a neural network. The system100 includes utterance shards 102 a-n, decoders 104 a-n, frame batches106-n, speech models 108 a-n, and a parameter server 110. Briefly, inthe system 100, the training data may be divided into a number ofutterance shards 102 a-n, and sequence training may be run with speechmodels 108 a-n for each of the utterance shards. The value of n may beany integer greater than one.

In some implementations, sequence training may run an independentdecoder 104 (e.g., any one of 104 a-n) in the input layer of each model108 (e.g., any one of 108 a-n) to generate on-the-fly lattices, and thencomputes an outer gradient. A centralized parameter server 110 may beconfigured to keep the current state of all model parameters for thespeech model replicas 108 a-n. In some implementations, the modelparameters may include the weights and biases of hidden layers of aneural network corresponding to a speech model of speech model replicas108 a-n. In some implementations, the centralized parameter server 110may be sharded across many parameter shards 112 a-112 k. The value of kmay be any integer greater than one.

In some implementations, the models 108 a-n may periodically updatetheir model parameters. For example, the models 108 a-n mayasynchronously request fresh values from the parameter server 110. Insome implementations, the models 108 a-n may send updates to theparameter server 110. In some implementations, the decoders 104 a-n mayrequest fresh values from the parameter server 110 to avoid stale outergradients. The functions performed by the system 100 can be performed byindividual computer systems or can be distributed across multiplecomputer systems over a network.

FIG. 1 also illustrates an example flow of data, shown in stages (A) to(F). Stages (A) to (F) may occur in the illustrated sequence, or theymay occur in a sequence that is different than in the illustratedsequence. In some implementations, one or more of the stages (A) to (F)may occur offline, where the system 100 may perform computations whenthe system 100 is not connected to a network.

During stage (A), the decoder 104 a obtains a set of training utterancesfrom the utterance shard 102 a. In some implementations, the trainingutterances may be one or more predetermined words spoken by the trainingspeakers that were recorded and accessible by the decoder 104 a. Eachtraining speaker may speak a predetermined utterance to a computingdevice, and the computing device may record an audio signal thatincludes the utterance. For example, each training speaker may beprompted to speak the training phrase “Hello Phone, show me the nearestpost office.” In some implementations, each training speaker may beprompted to speak the same training phrase multiple times. The recordedaudio signal of each training speaker may be transmitted to a computingsystem not shown here, and the computing system may collect the recordedaudio signals and divide the utterances into the utterance shards 102a-n. In other implementations, the various training utterances in theutterance shards 102 a-n may include utterances of different words.

During stage (B), the decoder 104 a obtains model parameters 114 a fromthe parameter server 110, and for each training utterance, the decoder104 a determines an output frame to be used for training the neuralnetwork of the speech model 108 a. As described in more details in FIG.1B, the decoder 104 a may determine an output frame representingacoustic characteristics of the utterance based on the reference scoreand the decoding score. In some implementations, the output frame may bea maximum mutual information (MMI) soft frame that is used to representtraining criterion for sequence training of the speech model 108 a. Forexample, the MMI soft frame corresponding to a training utterance may berepresented by a vector that includes speech content of the trainingutterance (e.g., log energy of the training utterance audio signal), oneor more soft-labels (e.g., outer gradients for individual HMM phoneticstates in a frame), and/or a confidence score associated with thedecoded training utterance.

During stage (C), the output frames from the decoder 104 a are batchedinto a frame batch 106 a. In some implementations, as described in moredetails in FIG. 1B, the output frames may be randomly shuffled beforethe frame batch 106 a is generated. The randomization of trainingutterances may provide a lower word error rate for a speech modeloptimized using a stochastic optimization process.

During stage (D), the frame batch 106 a is input to a speech model 108 afor training model parameters. During stage (E), the speech model 108 aobtains model parameters 114 a from the parameter server 110. Asdescribed in more details in FIG. 1C, the speech model 108 a may thendetermine updated model parameters based on the frame batch 106 a andthe model parameters 114 a. In some implementations, stochastic gradientdescent process may be used to optimize the updated model parameters.

In some implementations, the model parameters 114 a received by thespeech model 108 a may not be the most updated model parameters storedin the parameter server because the model parameters are being updatedby multiple speech model replicas 108 b-n asynchronously and inparallel. In some implementations, the speech model 108 a may apply anauxiliary function, which is an approximation to the original trainingobjective, to optimize the model parameters locally without taking intoconsiderations of further parameter updates by other speech models 108b-n.

During stage (F), the speech model 108 a outputs model parametergradients 116 a that represent differences between the model parameters114 a and updated model parameters that have been optimized by thespeech model 108 a. In some implementations, the speech model 108 a maysend the model parameter gradients 116 a to the parameter server 110,and the parameter server 110 may determine updated model parametersbased on the stored model parameters 114 a and the received modelparameter gradients 116 a. In some other implementations, the speechmodel 108 a may send the updated model parameters to the parameterserver 110, and the parameter server 110 may replace the modelparameters 114 a with the updated model parameters.

FIG. 1 further illustrates another example flow of data, shown in stages(A′) to (F′). The stages (A′) to (F′) are similar to stages (A) to (F)described above, but stages (A′) to (F′) are performed independentlyfrom stages (A) to (F). As used in this specification, to perform aspecific stage/operation independently from a second stage/operationrefers to performing the specific stage/operation without therequirement to check when the second stage/operation is being performed.In general, the system 100 in FIG. 1 is asynchronous in several aspects:the speech model replicas 108 a-n, parameter server shards 112 a-k, anddecoders 104 a-n may all run independently of each other. Since decodingand forced alignment to compute the gradient take time, asynchrony maybe introduced as those gradients are computed using stale and eveninconsistent model parameters. For evaluation, a number of additionalmodel replicas may be run to compute the statistics. In someimplementations, the amount of asynchrony depends on the number of modelreplicas and the batch size. For example, for 50 replicas, the averagestaleness of the outer gradients is around one minute, corresponding toa few hundred DNN update steps. Alternatively, a different number ofmodel replicas (e.g., 5, 100, 500, etc.) or the batch size (e.g., 100,1000, 10000, etc.) may be used.

During stage (A′), the decoder 104 n obtains a set of trainingutterances from the utterance shard 102 n. In some implementations, thedecoder 104 n may obtain the set of training utterances from theutterance shard 102 n independently and asynchronously from the otherdecoders 104 a-m.

During stage (B′), the decoder 104 n obtains model parameters 114 n fromthe parameter server 110, and for each training utterance, the decoder104 n determines an output frame to be used for training the neuralnetwork of the speech model 108 n. In some implementations, the decoder104 n may obtain the model parameters 114 n independently andasynchronously from the other decoders 104 a-m. For example, the decoder104 n may obtain the model parameters 114 n after the speech model 108 ahas updated the model parameters in the parameter server 110.

During stage (C′), the output frames from the decoder 104 n are batchedinto a frame batch 106 n. In some implementations, the decoder 104 n maygenerate the frame batch 106 n independently and asynchronously from theother decoders 104 a-m.

During stage (D′), the frame batch 106 n is input to a speech model 108n for training model parameters. During stage (E′), the speech model 108n obtains model parameters 114 n from the parameter server 110. In someimplementations, the speech model 108 n may obtain model parameters 114n from the parameter server 110 independently and asynchronously fromthe other speech models 108 a-m.

During stage (F′), the speech model 108 n outputs model parametergradients 116 n that represent differences between the model parameters114 n and updated model parameters that have been optimized by thespeech model 108 n. In some implementations, the speech model 108 n maysend the model parameter gradients 116 n to the parameter server 110,and the parameter server 110 may determine updated model parametersbased on the stored model parameters 114 n and the received modelparameter gradients 116 n. In some implementations the speech model 108n may send the model parameter gradients 116 n to the parameter server110 independently and asynchronously from the other speech models 108a-m.

FIG. 1B is a block diagram of an example system 100 for generating aframe batch for training a speech model. Briefly, the decoder 104obtains model parameters 114 and training utterances stored in theutterance shard 102, and generates a frame batch 106 used for training aspeech model 108. The decoder 104 includes a reference engine 122, adecoding engine 124, a frame generation engine 126, and a frameshuffling engine 128. The decoder 104 described here may be any one ofthe decoders 104 a-n described in FIG. 1A.

As used in this specification, an “engine” (or “software engine”) refersto a software implemented input/output system that provides an outputthat is different from the input. An engine can be an encoded block offunctionality, such as a library, a platform, a Software Development Kit(“SDK”), a software module, or an object.

In general, the reference engine 122 is configured to generate areference score 123 that represents the transcript truth of a trainingutterance. In some implementations, the reference engine 122 may obtainmodel parameters 114 from the parameter server 110. The reference engine122 may then identify one or more reference parameters associated withthe training utterance. In some implementations, the referenceparameters may include a true transcription of the training utterance.In some implementations, the reference parameters may include acousticfeatures of the training utterance. For example, the acoustic signal ofthe training utterance may be segmented into multiple time windows. AFast Fourier Transform (FFT) may be performed on the audio in eachwindow. The results of the FFT may be shown as time-frequencyrepresentations of the audio in each window. From the FFT data for awindow, features that are represented as an acoustic feature vector forthe window may be determined. The acoustic features may be determined bybinning according to filterbank energy coefficients, using amel-frequency ceptral component (MFCC) transform, using a perceptuallinear prediction (PLP) transform, or using other techniques. In someimplementations, the logarithm of the energy in each of various bands ofthe FFT may be used to determine acoustic features.

The reference engine 122 may then aligns a lattice corresponding to thetraining utterance and generates a reference lattice. The referencelattice represents the combination of words that form the truetranscriptions for the training utterance. Examples of word lattices areshown in FIGS. 6A and 6B. In some implementations, the reference latticemay be determined based on (i) the utterance, (ii) the speech modelparameters, and (iii) the reference parameters representing the truetranscription of the utterance.

The reference engine 122 may then determine a reference score for thetraining utterance. As described in more details in FIG. 1C, in someimplementations, the reference score may be a sub-component of an outergradient function. For example, the sub-component may include the stateoccupancy of a frame t within a training utterance u.

In general, the decoding engine 124 is configured to generate a decodingscore 125 that represents the decoded transcription of a trainingutterance. In some implementations, the decoding engine 124 may obtainmodel parameters 114 from the parameter server 110. The decoding engine124 may then identify one or more decoding parameters associated withthe training utterance. In some implementations, the decoding parametersmay include acoustic features of the training utterance. In someimplementations, the decoding parameters may represent a candidatetranscription of the utterance.

The decoding engine 124 may then generate a decoding latticecorresponding to the training utterance. The decoding lattice representsthe combination of words that form different candidate transcriptionsfor the training utterance. Examples of word lattices are shown in FIGS.6A and 6B. In some implementations, the decoding lattice may bedetermined based on (i) the utterance, (ii) the speech model parameters,and (iii) the decoding parameters representing the candidatetranscription of the utterance.

The decoding engine 124 may then determine a decoding score for thetraining utterance. As described in more details in FIG. 1C, in someimplementations, the decoding score may be a sub-component of an outergradient function.

In general, the frame generation engine 126 is configured to generate anoutput frame corresponding to the training utterance. In someimplementations, the output frame may be represented by a vector thatincludes speech content of the training utterance (e.g., log energy ofthe training utterance audio signal), one or more soft-labels (e.g.,outer gradients for individual HMM phonetic states in a frame), and/or aconfidence score associated with the decoded training utterance. Forexample, the output frame may include output gradients representingdifferences between the true transcription and the candidatetranscription of the utterance. In some implementations, the outergradient depends on the model parameters 114, and the outer gradientcannot be pre-computed. In some implementations, the outer gradients canbe negative and sum up to zero. In general, the gradient is exact for asingle, synchronous parameter update as is the case for SGD or batchoptimization. However, the gradient may only be approximate forasynchronous SGD (ASGD) because the outer gradients may not be updatedover multiple parameter updates.

In general, the frame shuffling engine 128 is configured to randomlyselect output frames to output a frame batch 106. The shuffling oftraining data for SGD has significant impact on the performance. Forexample, a frame batch 106 may have a batch size N, where N is aninteger greater than one. In some implementations, the batch size N maybe constant. Each slot in the batch may load a separate utterance andprocess the frames within an utterance one by one. When all frames of anutterance are consumed, another utterance is loaded. In someimplementations, the shuffling may happen at two different levels: abatch consists of frames each from different, random utterances, and alarge component of the randomization comes from running multiple modelreplicas independently and asynchronously on different subsets ofutterances. Note that shuffling does not involve shuffling the frameswithin an utterance because the context of the utterance needs to bepreserved in a speech recognition process using sequence decoding.

FIG. 1C is a block diagram of an example system 100 for training aneural network of a speech model. Briefly, the speech model 108 obtainsa frame batch 106 and model parameters 114, and generates modelparameter gradients 116 that represent updated model parameters. Thespeech model 108 includes a neural network 136, an auxiliary functionengine 132, and a model optimization engine 134. The speech model 108described here may be any one of the speech models 108 a-n described inFIG. 1A.

In general, the neural network 136 may serve as a speech model andindicate likelihoods that acoustic feature vectors represent differentphonetic units. The neural network 136 includes an input layer 142, anumber of hidden layers 144 a-144 m, and an output layer 146. The neuralnetwork 136 may receive acoustic feature vectors in the frame batch 106as input.

In general, the model optimization engine 134 is configured to train aneural network as a speech model. In some implementations, a speechmodel implemented using a neural network may be implemented as follows.Let X=x₁, . . . , x_(T) denote a sequence of T feature vectors and W aword sequence. According to the HMM assumption, the probability foracoustic models may be decomposed as shown in Equation (1), below:

$\begin{matrix}{{p\left( X \middle| W \right)} = {\sum\limits_{s_{1},\ldots,{s_{T} \in W}}{\prod\limits_{t = 1}^{T}{{p\left( x_{t} \middle| s_{t} \right)}{p\left( s_{t} \middle| s_{t - 1} \right)}}}}} & (1)\end{matrix}$

In Equation (1), the marginalization is over the HMM states s₁, . . . ,s_(T) representing the given word sequence W. In the hybrid modelingapproach, the emission probability may be represented asp(x|s)=p(s|x)p(x)/p(s) (Bayes rule). The state posterior p(s|x) may beestimated with a static classifier. The state prior p(s) is the relativestate frequency. The data likelihood p(x) does not depend on state s andthus can be ignored for decoding/lattice generation and forcedalignment. The model parameters may include the weights and biases ofone or more neural network layers, which may be estimated by minimizingthe cross-entropy (CE) on all utterances u and frames t as shown inEquation (2), below:

$\begin{matrix}{{F_{CE}(\theta)} = {{- \frac{1}{T}}{\sum\limits_{u}{\sum\limits_{t = 1}^{T_{u}}{\sum\limits_{s}{{l_{ut}(s)}\log\;{{p_{\theta}\left( s \middle| x_{ut} \right)}.}}}}}}} & (2)\end{matrix}$

Here,

$T = {\sum\limits_{u}T_{u}}$is the total number of frames. The targets may be set to l_(ut)(s)=δ(s,s_(ut)) for fixed state alignments s_(u1), . . . , s_(u)T_(u), where δdenotes the Kronecker delta.

An example training criterion for sequence training is maximum mutualinformation (MMI) as shown in Equation (3), below:

$\begin{matrix}{{F_{MMI}(\theta)} = {{- \frac{1}{T}}{\sum\limits_{u}{\log{\frac{{p_{\theta}\left( X_{u} \middle| W_{u} \right)}^{\kappa}{p\left( W_{u} \right)}}{\sum\limits_{W}{{p_{\theta}\left( X_{u} \middle| W \right)}^{\kappa}{p(W)}}}.}}}}} & (3)\end{matrix}$

In some implementations, a weak language model may be used for sequencetraining. For example, a unigram language model may be used and thelanguage model weight κ⁻¹ may be attached to the acoustic model. Thelogarithm diverges if the argument goes to zero, i.e., if the correctword sequence has zero probability in decoding. To avoid numericalissues with such utterances, the frame rejection heuristic may be used.For example, frames with zero state occupancy, γ_(ut) ^((den))(s)=0, maybe discarded. In some implementations, regularization (e.g.,12-regularization around the initial network) or smoothing (e.g., theH-criterion) may be used.

In some implementations, the gradient of the training criterion can bewritten as shown in Equation (4), below:

$\begin{matrix}{{\nabla{F_{MMI}(\theta)}} = {{- \frac{1}{T}}{\sum\limits_{u}{\sum\limits_{t = 1}^{T_{u}}{\sum\limits_{s}{{\kappa\left\lbrack {{\gamma_{\theta,{ut}}^{({num})}(s)} - {\gamma_{\theta,{ut}}^{({den})}(s)}} \right\rbrack} \times {\nabla\log}\;{p_{\theta}\left( s \middle| x_{ut} \right)}}}}}}} & (4)\end{matrix}$

In equation (4), γ_(θ,ut) ^((num/den))(s) denotes thenumerator/denominator state occupancy for utterance u and frame t.Relating to the chain rule terminology, and κ└γ_(θ,ut)^(num)(s)−γ_(θ,ut) ^(den)(s)┘ and ∇ log p_(θ)(s|x_(ut)) may be referredto as the outer and inner gradients, respectively. In someimplementations, the outer gradient depends on the model parameters θ,and the outer gradient cannot be pre-computed. In some implementations,the outer gradients can be negative and sum up to zero. In someimplementations, given the outer gradient encoded as targets, the innergradient may be computed in the same way as the gradient for F_(CE)(θ).In general, the gradient is exact for a single, synchronous parameterupdate as is the case for SGD or batch optimization. However, thegradient may only be approximate for ASGD because the outer gradientsmay not be updated over multiple parameter updates.

In general, the auxiliary function engine 132 is configured to apply anauxiliary function to locally optimize the neural network based onreceived model parameter 114 and outer gradients. The use of anauxiliary function allows more formal reasoning and justification of theuse of ASGD optimization for sequence training. Auxiliary functions areapproximations to the original training objective that may be simpler tooptimize. Here, “simpler” means that the auxiliary function may looklike a frame-level training criterion (e.g., Equation 2) that can be(partially) optimized with a stand-alone system such as the ASGD system100. In some implementations, the optimization of the total trainingcriterion may be performed iteratively by updating the auxiliaryfunction at the current parameter estimate θ′ and optimizing theauxiliary function to obtain a new estimate θ.

In some implementations, an auxiliary function may be a function thatmakes tangential contact with the training criterion at θ or lie in thehypograph of the training criterion. This type of auxiliary functionsmay be easy to construct although little can be said about convergence.Constructive and efficient lower bounds may be harder to find but maylead to generalized Expectation-Maximization, with stronger convergenceproperties, in particular convergence to a local optimum. In general,the tangential contact is only locally valid and requires frequentupdates of the outer gradient to guarantee stable convergence. Incontrast, a lower bound may be globally valid and expected to be lesssensitive to frequent updates of the outer gradient.

As an example, the following auxiliary function, as shown in Equation(5) below, may be used for F_(MMI) (Equation 3)

$\begin{matrix}{{T_{MMI}\left( {\theta^{\prime},\theta} \right)} = {{{- \frac{1}{T}}{\sum\limits_{u}{\sum\limits_{t = 1}^{T_{u}}{\sum\limits_{s}{{l_{\theta^{\prime},{ut}}(s)}\log\;{p_{\theta}\left( s \middle| x_{ut} \right)}}}}}} + {F_{MMI}\left( \theta^{\prime} \right)}}} & (5)\end{matrix}$

with l_(θ′,ut)(s)=κ└γ_(θ,ut) ^((num))(s)−γ_(θ,ut) ^((den))(s)┘ toenforce ∇T_(MMI)(θ′, θ)|_(θ=θ′)=∇=F_(MMI)(θ′, θ)|_(θ=θ′).

After the training criteria are satisfied, the model optimization engine134 may determine updated model parameters. The speech model 108 outputsmodel parameter gradients 116 that represent differences between themodel parameters 114 and updated model parameters that have beenoptimized by the speech model 108.

FIG. 2 is a flow chart illustrating an example process 200 fordetermining a frame for training a speech model. The process 200 may beperformed by data processing apparatus, such as the decoder 104described above or another data processing apparatus.

The system obtains a training utterance (202). In some implementations,the training utterance may be one or more predetermined words spoken bya training speaker that were recorded and accessible by the system. Insome implementations, the various training utterances in an utteranceshard may include utterances of different words.

The system determines a reference score for the training utterance(204). As described in more details in FIG. 3 below, the reference scoremay be a sub-component of an outer gradient function that represents thetranscript truth of the training utterance.

The system determines a decoding score for the training utterance (206).As described in more details in FIG. 4 below, the decoding score may bea sub-component of an outer gradient function that represents thecandidate transcript of the training utterance.

The system determines a frame for training a neural network model (208).In some implementations, the output frame may be a maximum mutualinformation (MMI) soft frame that is used to represent trainingcriterion for sequence training of the speech model. For example, theMMI soft frame corresponding to the training utterance may berepresented by a vector that includes speech content of the trainingutterance (e.g., log energy of the training utterance audio signal), oneor more soft-labels (e.g., outer gradients for individual HMM phoneticstates in a frame), and/or a confidence score associated with thedecoded training utterance.

FIG. 3 is a flow chart illustrating an example process 300 fordetermining a reference score in a decoder. The process 300 may beperformed by data processing apparatus, such as the decoder 104described above or another data processing apparatus.

The system obtains a training utterance (302). The system also obtainsmodel parameters from a parameter server (304). In some implementations,the model parameters may include the weights and biases of hidden layersof a neural network corresponding to a speech model.

The system obtains one or more reference parameters (306). In someimplementations, the one or more reference parameters may represent atrue transcription of the utterance. In some implementations, thereference parameters may include a true transcription of the trainingutterance. In some implementations, the reference parameters may includeacoustic features of the training utterance.

The system aligns reference lattice (308). In some implementations, thereference lattice may be determined based on (i) the utterance of thefirst training utterances, (ii) the one or more first neural networkparameters, and (iii) the one or more reference parameters representingthe true transcription of the utterance. The system then determines areference score (310). In some implementations, the reference score maybe a sub-component of an outer gradient function that represents thetranscript truth of the training utterance.

FIG. 4 is a flow chart illustrating an example process 400 fordetermining a decoding score in a decoder. The process 400 may beperformed by data processing apparatus, such as the decoder 104described above or another data processing apparatus.

The system obtains a training utterance (402). The system also obtainsmodel parameters from a parameter server (404). The system then obtainsdecoding parameters (406). In some implementations, the decodingparameters may include acoustic features of the training utterance. Insome implementations, the decoding parameters may represent a candidatetranscription of the utterance.

The system obtains a decoding lattice (408). The decoding latticerepresents the combination of words that form different candidatetranscriptions for the training utterance. In some implementations, thedecoding lattice may be determined based on (i) the utterance, (ii) thespeech model parameters, and (iii) the decoding parameters representingthe candidate transcription of the utterance.

The system determines a decoding score (410). In some implementations,the decoding score may be a sub-component of an outer gradient functionthat represents the candidate transcript of the training utterance.

FIG. 5 is a flow chart illustrating an example process 500 forgenerating model parameter gradients for a speech model. The process 500may be performed by data processing apparatus, such as the speech model108 described above or another data processing apparatus.

The system obtains utterance batch (502). In some implementations, afirst sequence-training speech model may obtain a first batch oftraining frames that represent speech features of first trainingutterances. In some implementations, a second sequence-training speechmodel may obtain a second batch of training frames that represent speechfeatures of second training utterances, where the obtaining of thesecond batch of training frames by the second sequence-training speechmodel is independent of the obtaining of the first batch of trainingframes by the first sequence-training speech model.

In some implementations, before the first sequence-training speech modelobtains the first batch of training frames, the first sequence-trainingspeech model may obtain an initial batch of training frames, where eachtraining frame represents a sequence of utterances spoken by a trainingspeaker. The first sequence-training speech model may pseudo-randomlyselect candidate training frames from the initial batch of trainingframes. The first sequence-training speech model may generate the firstbatch of training frames using the pseudo-randomly selected candidatetraining frames.

The system obtains model parameters from a parameter server (504). Insome implementations, the first sequence-training speech model mayobtain one or more first neural network parameters. In someimplementations, the second sequence-training speech model may obtainone or more second neural network parameters, where the obtaining of thesecond neural network parameters may be independent of (i) the obtainingof the first neural network parameters by the first sequence-trainingspeech model and (ii) the determining of the one or more optimized firstneural network parameters by the first sequence-training speech model.

The system obtains an auxiliary function (506). In some implementations,the auxiliary function may represent an approximation of a trainingobjective function of the first sequence-training speech model. Forexample, the training objective function may be a maximum mutualinformation (MMI) objective function, a minimum phone error (MPE)objective function, or a state-level minimum Bayes risk objectivefunction.

The system optimizes a localized neural network (508). In someimplementations, the first sequence-training speech model may determineone or more optimized first neural network parameters based on (i) thefirst batch of training frames and (ii) the one or more first neuralnetwork parameters. In some implementations, the first sequence-trainingspeech model or the second sequence-training speech model may determinethe one or more optimized neural network parameters using an auxiliaryfunction.

In some implementations, the second sequence-training speech model maydetermine one or more optimized second neural network parameters basedon (i) the second batch of training frames and (ii) the one or moresecond neural network parameters, where the determining of the one ormore optimized second neural network parameters by the secondsequence-training speech model may be independent of the determining ofthe one or more optimized first neural network parameters by the firstsequence-training speech model.

In some implementations, the first sequence-training speech model maydetermine one or more optimized first neural network parameters byupdating a last hidden layer of a neural network of the first sequencetraining speech model, using (i) the first batch of training frames and(ii) the one or more first neural network parameters to determine theone or more optimized first neural network parameters.

In some implementations, the first sequence-training speech model maydetermine one or more optimized first neural network parameters byupdating a plurality of hidden layers of a neural network of the firstsequence training speech model, using (i) the first batch of trainingframes and (ii) the one or more first neural network parameters todetermine the one or more optimized first neural network parameters.

The system generates model parameter gradients that representdifferences between the model parameters and updated model parametersthat have been optimized (510). In some implementations, the system maysend the model parameter gradients to a parameter server. In some otherimplementations, the system may send the updated model parameters to theparameter server.

FIG. 6A is an example of a word lattice 600 that may be provided by aspeech recognizer system. The word lattice 600 represents multiplepossible combinations of words that may form different candidatetranscriptions for an utterance.

The word lattice 600 includes one or more nodes 602 a-g that correspondto the possible boundaries between words. The word lattice 600 includesmultiple edges 604 a-l for the possible words in the transcriptionhypotheses that result from the word lattice 600. In addition, each ofthe edges 604 a-l can have one or more weights or probabilities of thatedge being the correct edge from the corresponding node. The weights aredetermined by the speech recognizer module system and can be based on,for example, a confidence in the match between the speech data and theword for that edge and how well the word fits grammatically and/orlexically with other words in the word lattice 600.

For example, initially, the most probable path through the word lattice600 may include the edges 604 c, 604 e, 604 i, and 604 k, which have thetext “we're coming about 11:30.” A second best path may include theedges 604 d, 604 h, 604 j, and 604 l, which have the text “deer huntingscouts 7:30.”

Each pair of nodes may have one or more paths corresponding to thealternate words in the various candidate transcriptions. For example,the initial most probable path between the node pair beginning at thenode 602 a and ending at the node 602 c is the edge 604 c “we're”. Thispath has alternate paths that include the edges 604 a-b “we are” and theedge 604 d “deer”.

FIG. 6B is an example of a hierarchical word lattice 650 that may beprovided by a speech recognizer system. The word lattice 650 includesnodes 652 a-l that represent the words that make up the variouscandidate transcriptions for an utterance. The edges between the nodes652 a-l show that the possible candidate transcriptions include: (1) thenodes 652 c, 652 e, 652 i, and 652 k “we're coming about 11:30”; (2) thenodes 652 a, 652 b, 652 e, 652 i, and 652 k “we are coming about 11:30”;(3) the nodes 652 a, 652 b, 652 f, 652 g, 652 i, and 652 k “we are comeat about 11:30”; (4) the nodes 652 d, 652 f, 652 g, 652 i, and 652 k“deer come at about 11:30”; (5) the nodes 652 d, 652 h, 652 j, and 652 k“deer hunting scouts 11:30”; and (6) the nodes 652 d, 652 h, 652 j, and652 l “deer hunting scouts 7:30”.

Again, the edges between the nodes 652 a-l may have associated weightsor probabilities based on the confidence in the speech recognition andthe grammatical/lexical analysis of the resulting text. In this example,“we're coming about 11:30” may currently be the best hypothesis and“deer hunting scouts 7:30” may be the next best hypothesis. One or moredivisions 654 a-d can be made in the word lattice 650 that group a wordand its alternates together. For example, the division 654 a includesthe word “we're” and the alternates “we are” and “deer”. The division654 b includes the word “coming” and the alternates “come at” and“hunting”. The division 654 c includes the word “about” and thealternate “scouts” and the division 654 d includes the word “11:30” andthe alternate “7:30”.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved.

Embodiments and all of the functional operations described in thisspecification may be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments may be implemented asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer-readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable-medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter affecting a machine-readable propagated signal, or a combinationof one or more of them. The computer-readable medium may be anon-transitory computer-readable medium. The term “data processingapparatus” encompasses all apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus mayinclude, in addition to hardware, code that creates an executionenvironment for the computer program in question, e.g., code thatconstitutes processor firmware, a protocol stack, a database managementsystem, an operating system, or a combination of one or more of them. Apropagated signal is an artificially generated signal, e.g., amachine-generated electrical, optical, or electromagnetic signal that isgenerated to encode information for transmission to suitable receiverapparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any form of programminglanguage, including compiled or interpreted languages, and it may bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program may be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programmay be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Moreover, a computer may be embedded inanother device, e.g., a tablet computer, a mobile telephone, a personaldigital assistant (PDA), a mobile audio player, a Global PositioningSystem (GPS) receiver, to name just a few. Computer readable mediasuitable for storing computer program instructions and data include allforms of non-volatile memory, media and memory devices, including by wayof example semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, embodiments may be implementedon a computer having a display device, e.g., a CRT (cathode ray tube) orLCD (liquid crystal display) monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user may provide input to the computer. Other kinds ofdevices may be used to provide for interaction with a user as well; forexample, feedback provided to the user may be any form of sensoryfeedback, e.g., visual feedback, auditory feedback, or tactile feedback;and input from the user may be received in any form, including acoustic,speech, or tactile input.

Embodiments may be implemented in a computing system that includes aback end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user may interact with animplementation of the techniques disclosed, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system may be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations, but rather as descriptions of featuresspecific to particular embodiments. Certain features that are describedin this specification in the context of separate embodiments may also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment mayalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination may in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems maygenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments have been described. Other embodiments arewithin the scope of the following claims. For example, the actionsrecited in the claims may be performed in a different order and stillachieve desirable results.

What is claimed is:
 1. A method performed by one or more computers, themethod comprising: obtaining, by the one or more computers, a replica ofa neural network to be trained; updating, by the one or more computers,parameters of the replica of the neural network based on data indicatingmodel parameters determined based on training operations for one or moreother replicas of the neural network; after updating the parameters ofthe replica of the neural network, performing, by the one or morecomputers, one or more training operations for the replica of the neuralnetwork based on a subset of training data for training the neuralnetwork; and sending, by the one or more computers, data indicatingresults of the one or more training operations.
 2. The method of claim1, wherein the subset of training data comprises data indicating of oneor more utterances.
 3. The method of claim 1, wherein performing the oneor more training operations comprises performing the one or moretraining operations asynchronously with respect to training operationsfor the one or more other replicas of the neural network.
 4. The methodof claim 1, wherein sending the data indicating results of the one ormore training operations comprises sending the data to indicatingresults of the one or more training operations to a server.
 5. Themethod of claim 1, wherein sending the data indicating results of theone or more training operations comprises sending data indicating one ormore model parameter updates.
 6. The method of claim 1, wherein sendingthe data indicating results of the one or more training operationscomprises sending data indicating one or more gradients.
 7. The methodof claim 1, wherein the neural network is a speech recognition model. 8.The method of claim 1, wherein the neural network is an acoustic model.9. The method of claim 1, wherein the neural network is a speech modeltrained to indicate likelihoods that acoustic feature vectors representdifferent phonetic units.
 10. The method of claim 1, wherein the subsetof training data is pseudo-randomly selected from the training data. 11.A system comprising: one or more computers; and one or morecomputer-readable media storing instructions that, when executed by theone or more computers, cause the one or more computers to performoperations comprising: obtaining, by the one or more computers, areplica of a neural network to be trained; updating, by the one or morecomputers, parameters of the replica of the neural network based on dataindicating model parameters determined based on training operations forone or more other replicas of the neural network; after updating theparameters of the replica of the neural network, performing, by the oneor more computers, one or more training operations for the replica ofthe neural network based on a subset of training data for training theneural network; and sending, by the one or more computers, dataindicating results of the one or more training operations.
 12. Thesystem of claim 11, wherein the subset of training data comprises dataindicating of one or more utterances.
 13. The system of claim 11,wherein performing the one or more training operations comprisesperforming the one or more training operations asynchronously withrespect to training operations for the one or more other replicas of theneural network.
 14. The system of claim 11, wherein sending the dataindicating results of the one or more training operations comprisessending the data to indicating results of the one or more trainingoperations to a server.
 15. The system of claim 11, wherein sending thedata indicating results of the one or more training operations comprisessending data indicating one or more model parameter updates.
 16. Thesystem of claim 11, wherein sending the data indicating results of theone or more training operations comprises sending data indicating one ormore gradients.
 17. The system of claim 11, wherein the neural networkis a speech recognition model.
 18. The system of claim 11, wherein theneural network is an acoustic model.
 19. The system of claim 11, whereinthe neural network is a speech model trained to indicate likelihoodsthat acoustic feature vectors represent different phonetic units. 20.One or more non-transitory computer-readable media storing instructionsthat, when executed by the one or more computers, cause the one or morecomputers to perform operations comprising: obtaining, by the one ormore computers, a replica of a neural network to be trained; updating,by the one or more computers, parameters of the replica of the neuralnetwork based on data indicating model parameters determined based ontraining operations for one or more other replicas of the neuralnetwork; after updating the parameters of the replica of the neuralnetwork, performing, by the one or more computers, one or more trainingoperations for the replica of the neural network based on a subset oftraining data for training the neural network; and sending, by the oneor more computers, data indicating results of the one or more trainingoperations.